Workflow-based Services: Infrastructure for Scientific Applications

The way scientists work in traditional sciences has changed drastically in recent years. Computer science is increasingly supporting them in performing and analyzing their experiments. Today, obtaining the raw data from instruments is merely the first step. No longer is the data analyzed on paper or with simple computational tools. Instead, massive amounts of data obtained raw from instruments are processed in complex and long-running computational pipelines. This trend of supporting traditional sciences with computational tools has lead to a significant speedup in executing experiments and has also enabled experiments which would not have been possible before. Scientists increasingly depend on adequate infrastructure to process experiment data using complex computational pipelines and to manage the plethora of data used and produced by them. Such computational pipelines are typically modeled as workflows and so this trend consequently challenges the current infrastructure for executing workflows as well as the infrastructure to manage the resulting data deluge. The work presented in this book addresses some of the challenges arising from this trend. A first challenge this book addresses is the sharing of scientific computations. Publishing a methodology for others to use is no longer possible with the scientific computations or workflows people develop today. Implementing such workflows and setting up the appropriate environment is far too difficult and consequently such computations or workflows must be shared. State-of-the-art approaches use proprietary interfaces to make such workflows available as services to be used by others. Proprietary interfaces, however, make it difficult to integrate such computations and we therefore use standardized interfaces. We describe in detail how we have mapped a workflow to such a standardized interface and report on an efficient implementation. The second challenge this book examines is how to provide adequate infrastructure for scientific workflows. One considerable advantage of scientific workflows is that they can be parametrized and one workflow can be run several thousand times for processing vast numbers of raw data sets obtained from instruments. Such large numbers of executions mandate the execution engine to be distributed in order to scale. However, configuring such a distributed engine is very difficult. Also, if it is configured optimally for just one type of workload, it may not be suitable for another. For this reason, we have added an autonomic controller to a distributed workflow execution engine. This controller monitors the behavior of the engine and adapts the configuration on the fly to serve the current workload best, adding self-configuration and self-optimization properties. The controller also heals the system in case of failures, thereby adding self-healing properties.